Total number of cases of cholera in each governate in Yemen

Daily number of new cases of cholera in each governate in Yemen

Total number of deaths from cholera in each governate in Yemen

Cholera Deaths Table

Cholera Cases Table

Total number of cases of cholera in each governate in Yemen

Daily number of new cases of cholera in each governate in Yemen

Total number of deaths from cholera in each governate in Yemen

Cholera Deaths Table

Cholera Cases Table

Total number of cases of cholera in each governate in Yemen

Daily number of new cases of cholera in each governate in Yemen

Total number of deaths from cholera in each governate in Yemen

Cholera Deaths Table

Cholera Cases Table

Cholera Deaths Table

Cholera Cases Table

hdx_resource_csv

The new function hdx_resource_csv will take the results of hdx_resource_list, and return a new dataframe. This will have three columns:

• identifier a title merged from the package and dataset titles
• location the url where the csv was downloaded from
• csv a nested dataframe downloaded from the location url

hdx_resource_csv provides a nested dataframe column to allow a sensible output when you download multiple csvs all with different columns in one go.

When you want to extract a particular csv to work with, you can use dplyr::filter() and tidyr::unnest() to get at it. You could just unnest without filtering, but when every csv has different column titles, the output is a bit messy.

Example:

library(tidyverse)
library(hdxr)
hdx_connect()
datasets <- hdx_package_search(term = "141121-sierra-leone-health-facilities") %>%
  hdx_resource_list() %>%
  hdx_resource_csv()

The above would give us a dataframe that looks like:

you can then select what you want with filter and unnest it:

sierra_leone_healthsites_sbtf_sle_health <- datasets %>%
  filter(dataset_identifier == "sierra-leone-healthsites_sbtf-sle-health") %>%
  unnest()

hdx_connect()

This will use the ckanr package to connect to the HDX ckan server.

# This creates a function to connect to the hdx server
hdx_connect <- function(){
ckanr_setup(url = "http://data.humdata.org/")
}

hdx_list()

This function takes one argument, limit. It will return a list of HDX packages, depending on the limit set. There’s currently almost 5000 packages.

# This function will create a data frame of the length set in x, listing the packages in x
hdx_list <- function(limit){
package_list(as = 'table', limit = limit) %>%
as_data_frame(.)
}

hdx_package_search()

So after seeing roughly which package you want to look at, either through the package list, or browsing on the website, you would search for it through this function.

This will return a dataframe, with some nested lists, and nested dataframes. It’s reasonably easy to browse to then work out exactly which package you want.

hdx_package_search <- function(term){
package_search(q = term, as = "json") %>%
    jsonlite::fromJSON(., flatten = TRUE) %>%
    .$result %>%
    .$results 
}

hdx_resource_list()

To see the exact resources available in an easier-to-read dataframe they need unnesting. When this function is used on the results of hdx_package_search(), it will extract a resources dataframe, then left join the results onto the original dataframe provided to it.

# This function will take the results of a package_search and extract the resources, it will then link those resources to the results from package search giving a new data frame
hdx_resource_list <- function(package){
package$resources %>%
    as.data.frame(.) %>%
    left_join(package, ., by = c("id" = "package_id")) %>%
    select(-resources)
}

Example: Exploring the ACLED Conflict Data for Africa

In this example I am going to identify the package “ACLED Conflict Data for Africa (Realtime - 2017)” here. It has an excel file, and a zipped csv. I’ll then identify the resource for the unzipped excel file.

Once we have that, we can use httr to download the file, and readxlto load it in.

library(ckanr)
library(tidyverse)
library(jsonlite)
library(readxl)
library(httr)

# First we connect to HDX

hdx_connect()

# We can list all packages available for us

hdx_list(5000)

## # A tibble: 4,905 x 1
##                                                                          value
##                                                                          <chr>
##  1                                       141121-sierra-leone-health-facilities
##  2                                      160516-ecuador-earthquake-4w-1st-round
##  3                                                      160523-ocha-4w-round-2
##  4                                                     160625-hrrp-4w-national
##  5 1999-2013-tally-of-internaly-displaced-persons-resulting-from-natural-disas
##  6                                                                  2011-nepal
##  7                                       2012-census-tanzania-wards-shapefiles
##  8                                        2014-2015-food-security-ipc-analysis
##  9                         2014-nutrition-smart-survey-results-and-2015-trends
## 10                                 2015-humanitarian-needs-overview-indicators
## # ... with 4,895 more rows

# We then search for the package we want, and use dplyr to filter it to the exact package

hum_data_packages <- hdx_package_search("ACLED Conflict Data for Africa") %>%
  filter(title == "ACLED Conflict Data for Africa (Realtime - 2017)")

# We then expand our resources from the search result, and look for the unzipped excel file

hum_data_resources <- hdx_resource_list(hum_data_packages) %>%
  filter(format == "XLSX")

url <- hum_data_resources$hdx_rel_url
GET(url, write_disk("dataset.xlsx", overwrite=TRUE))

## Response [http://www.acleddata.com/wp-content/uploads/2017/06/ACLED-All-Africa-File_20170101-to-20170617.xlsx]
##   Date: 2017-07-03 12:21
##   Status: 200
##   Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
##   Size: 1.86 MB
## <ON DISK>  C:\Users\callu\Dropbox\blog\content\post\dataset.xlsx

ACLED_CONFLICT_DATA <- read_excel("dataset.xlsx", col_names = TRUE)
rm(url)

Deprivation in Scotland by Health Boards

So looking at this, brighter colours equal lower average levels of deprivation. A distribution towards the left shows a healthboard with a greater proportion of deprived datazones.

Using the geofacet package, we swap out a normal facet_wrap() for facet_geo(), and tell it what layout we want grid=("nhs_scot_grid"). The only problem is that this grid has “Western Isles” saved as “Western Isle”, so we have to rename out own data to match this. Now our health boards are placed in roughly a geographical order.

data_simd_ranks %>%
  select(DataZone = DZ, HBname) %>%
  left_join(map_scot, .) %>%
  group_by(HBname) %>%
  mutate(median_deprivation = median(Rank)) %>%
  ungroup() %>%
  ggplot() +
  geom_histogram(aes(x = Rank, y = ..ncount.., fill = median_deprivation), binwidth = 100) +
  facet_geo(~HBname, grid = "nhs_scot_grid") +
  labs(title = "Deprivation in Health Boards in Scotland",
       x = "Relative deprivation of data zones (left = more deprived neighbourhoods)",
       y = "Proportion of data zones",
       caption = "Darker colour shows increased average deprivation of healthboard") + 
  theme_bw() +
  theme(legend.position = "none") 

Glasgow versus Edinburgh

In the plot above, the west coast / east coast divide is looking pretty big in the central belt of Scotland. The plot below shows this even more. Glasgows neighbourhoods are massively skewed towards some of the most deprived parts of Scotland, whereas in Edinburgh we see the opposite. Obviously Glasgow still has some very well off parts, and Edinburgh has some deprived areas, but in this graph they seem to be polar opposites.

city_data <- data_simd_ranks %>%
  select(DataZone = DZ, LAname) %>%
  left_join(map_scot, .) %>%
  group_by(LAname) %>%
  filter(LAname == "Glasgow City" | LAname == "City of Edinburgh") %>%
  mutate(median_deprivation = median(Rank)) %>%
  ungroup()

city_data$LAname <- parse_factor(city_data$LAname, levels = c("Glasgow City", "City of Edinburgh"))

  ggplot(city_data) +
  geom_histogram(aes(x = Rank, y = ..ncount.., fill = median_deprivation), binwidth = 100) +
  facet_wrap(~LAname) +
  labs(title = "Deprivation in Glasgow and Edinburgh",
       x = "Relative deprivation of data zones",
       y = "Proportion of data zones",
       caption = "Distribution to the left shows increased deprivation in city, darker colour shows increased average deprivation of city") + 
  theme_bw() +
  theme(legend.position = "none")